Geographic information systems using digital watermarks

ABSTRACT

The present invention relates to systems and methods for managing images or video and related information. In one embodiment, geographic information system (GIS) layer information is registered or aligned to an image or video using digital watermark information embedded within the image or video. Other embodiments are provided as well.

RELATED APPLICATION DATA

This patent application is a continuation of U.S. patent applicationSer. No. 10/423,834, filed Apr. 25, 2003 (published as US 2004-0008866A1). The 10/423,834 application claims the benefit of U.S. ProvisionalPatent Application Nos. 60/376,720, filed Apr. 29, 2002 and 60/383,474,filed May 23, 2002. The Ser. No. 10/423,834 application is also acontinuation in part of U.S. patent application Ser. No. 09/800,093,filed Mar. 5, 2001 (published as US 2002-0124171 A1). The Ser. No.10/423,834 application is also a continuation in part of U.S. patentapplication Ser. No. 10/002,954, filed Oct. 23, 2001 (published as US2002-0122564 A1), which is a continuation in part of U.S. patentapplication Ser. No. 09/800,093, filed Mar. 5, 2001. The Ser. No.10/002,954 application also claims the benefit of U.S. ProvisionalPatent Application Nos. 60/284,163, filed Apr. 16, 2001 and 60/284,776,filed Apr. 18, 2001. The Ser. No. 10/423,834 application is also acontinuation in part of U.S. patent application Ser. No. 10/100,233(published as US 2002-0154144 A1), filed Jun. 5, 2002, and acontinuation in part of U.S. patent application Ser. No. 09/858,336,filed May 15, 2001 (published as US 2002-0124024 A1). The 10/100,233application claims the benefit of U.S. Provisional Patent ApplicationNo. 60/284,776. The Ser. No. 10/423,834 application is also related toSer. No. 09/833,013 (published as US 2002-0147910 A1), PCT applicationPCT/US02/06858 (published as WO 02/071685) and U.S. patent applicationSer. No. 10/423,489 (published as US 2004-0046774 A1). Each of thesepatent documents is hereby incorporated by reference.

FIELD OF THE INVENTION

The present invention relates to image management and processing, and isparticularly illustrated in the context of management of satellite andother aerial imagery.

BACKGROUND AND SUMMARY OF THE INVENTION

Acquisition of aerial imagery traces its history back to the Wrightbrothers, and is now commonly performed from satellite and space shuttleplatforms, in addition to aircraft.

While the earliest aerial imagery relied on conventional filmtechnology, a variety of electronic sensors are now more commonly used.Some collect image data corresponding to specific visible, UV or IRfrequency spectra (e.g., the MultiSpectral Scanner and Thematic Mapperused by the Landsat satellites). Others use wide band sensors. Stillothers use radar or laser systems (sometimes stereo) to sensetopological features in three dimensions. Other types of imagecollection rely on electro-optical panchromatic (grayscale),multi-spectral (less than 20 bands) and hyper-spectral (20 bands ormore). Some satellites can even collect ribbon imagery (e.g., araster-like, 1-dimensional terrestrial representation, which is piecedtogether with other such adjacent ribbons).

The quality of the imagery has also constantly improved. Some satellitesystems are now capable of acquiring image and topological data having aresolution of less than a meter. Aircraft imagery, collected from loweraltitudes, provides still greater resolution.

A huge quantity of aerial imagery is constantly being collected.Management and coordination of the resulting large data sets is agrowing problem. Integrating the imagery with related information isalso a problem.

In accordance with one aspect of the present invention, digitalwatermarking technology is employed to help manage such imagery andrelated information, among other benefits. In another aspect, a digitalwatermark conveys information that is used to register or aligngeographic information system (GIS) layers with a corresponding imagelocation, perhaps after the image has been distorted.

The foregoing and additional features and advantages of the presentinvention will be even more readily apparent from the following detaileddescription with reference to the following figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an imagery distribution channel.

FIG. 2 illustrates license compliance facilitated via digitalwatermarks.

FIGS. 3-7 illustrate an image registration process, including an eventbroadcast feature; and FIG. 8 illustrates a network terminal that hasreceived an alert notification in the FIG. 7 environment.

FIG. 9 a illustrates an aerial image depicting a river section; and FIG.9 b illustrates a geographic information system layer corresponding tothe river shoreline of FIG. 9 a.

DETAILED DESCRIPTION

For expository convenience, the following section focuses on satelliteand aerial “imagery” to illustrate the principles of the invention. Theprinciples of the invention, however, are equally applicable to otherforms of captured aerial data and other topographic/mapping information.Accordingly, the term “image” should be used to encompass all such otherdata sets, and the term “pixel” should be construed to encompasscomponent data from such other data sets.

When new aerial imagery is received, it is generally necessary toidentify the precise piece of earth to which it corresponds. Thisoperation, termed “georeferencing” or “geocoding,” can be a convolutedart and science.

In many systems, the georeferencing begins with a master referencesystem (e.g., latitude and longitude) that takes into account theearth's known deformities from a sphere. Onto this reference system theposition of the depicted region is inferred, e.g., by consideration ofthe satellite's position and orientation (ephemeris data), opticalattributes of the satellite's imaging system (e.g., resolution,magnification, etc.), and models of the dispersion/refraction introducedby the earth's atmosphere.

In applications where precise accuracy is required, the foregoing,“ephemeris,” position determination is refined by comparing features inan image with the placement of known features on the earth's surface(e.g., buildings and other man-placed objects, geological features,etc.) and compensating the georeference determination accordingly. Thus,for example, if the actual latitude and longitude of a building is known(e.g., by measurement from a ground survey—“ground truth”), and thecorresponding latitude and longitude of that building as indicated inthe georeferenced satellite imagery is different, the reference systemapplied to the satellite data can be altered to achieve a match.(Commonly, three or more such ground truth points are used so as toassure accurate correction.) Of course these processes can involve bothmanual and automated steps.

Regardless of the georeferencing techniques used, once determined, adigital watermark can be used to convey geo-coordinates (or othergeoreferencing information—hereafter both referred to asgeo-coordinates). The geo-coordinates can be as simple as longitude andlatitude, or can be more finely referenced, e.g., with a geovector asdescribed in our related applications. The georeferencing may alsoinclude image scale and/or orientation. A digital watermark can includethe geo-coordinates as a message or payload. Or the digital watermarkcan include an identifier or index that is used to access or interrogatea geo-coordinate database.

Digital watermarking is a process for modifying physical or electronicmedia to embed a machine-readable code into the media. The media may bemodified such that the embedded code is imperceptible or nearlyimperceptible to the user, yet may be detected through an automateddetection process. Most commonly, digital watermarking is applied tomedia signals such as images, audio signals, and video signals. However,it may also be applied to other types of media objects, includingdocuments (e.g., through line, word or character shifting), software,multi-dimensional graphics models, and surface textures of objects.

Digital watermarking systems typically include two primary components:an encoder that embeds the watermark in a host media signal, and adecoder that detects and reads the embedded watermark from a signalsuspected of containing a watermark (a suspect signal). The encoderembeds a watermark by altering the host media signal. The readingcomponent analyzes a suspect signal to detect whether a watermark ispresent. In applications where the watermark encodes information, thereader extracts this information from the detected watermark.

Several particular watermarking techniques have been developed. Thereader is presumed to be familiar with the literature in this field.Particular techniques for embedding and detecting imperceptiblewatermarks in media signals are detailed in the assignee's U.S. patentapplication Ser. No. 09/503,881 (now U.S. Pat. No. 6,614,914) and U.S.Pat. No. 6,122,403, which are each herein incorporated by reference. Ofcourse, there are other suitable digital watermarking techniques thatcan be interchanged with some aspects of the present invention.

One inventive aspect of the present invention is to randomly orpseudo-randomly vary watermark identifiers used by a particular imagecapture device or watermarking embedder. Or a block of identifiers canbe pseudo-randomly generated and then delivered to the watermarkembedder. Varying the identifier will help prevent would be pirates fromdetermining which identifiers originate from a particular node orembedding station.

Digital watermarking an aerial image offers a host of advantages. Awatermark can be used to convey both static information and dynamicinformation. Examples of static information are the geo-coordinates ofthe image depicted in the image, the time and day the image wascaptured, and a source type indicator. The source type indicator canindicate the imaging source, e.g., governmental or commercial, theimaging sensor used to capture the image, or even the aerial imagecapture platform and characteristics, e.g., satellite, unmannedaircraft, etc. The source type indicator can be used to regulate accessto the image. Of course the static information can be conveyed directlyby a digital watermark or can be accessed via a data record associatedwith or pointed to by the digital watermark identifier. Examples ofdynamic information are analyst reports, current weather conditions inthe depicted area, notes, current news, video, audio, related images andinformation, etc.

In our related applications, identified-above, we disclose techniques bywhich a depicted location area is embedded with a digital watermark soas to unique identify that particular location of a map, image orphotograph (e.g., to identify geocoordinates associated with a depictedarea, building, road, lake, etc.). An image can be segmented into blocks(or areas). A digital watermark is embedded in each block (or area) toconvey (or point to) information that identifies the respective block'scenter geo-location, boundaries, corners, or object coordinates, etc. Aposition within the image is determined by reading the digital watermarkembedded within a corresponding block.

In some implementations, a block can be further segmented intosub-blocks. Each of the sub-blocks preferably includes the identifyinginformation for its parent block. This sub-blocking technique canprovide increased watermark detection robustness. Essentially, whentaking an image as a whole, the digital watermark(s) varies from blockto block within the image to accommodate the unique identifiers. Theblock (or area) size can be finely referenced, e.g., a digital watermarkcan be embedded to uniquely identify a block of pixels and even an arearepresented by a single pixel.

In other implementations, the image includes an embedded identifier,which is redundantly embedded throughout the image. The identifier linksback to metadata, e.g., to uniquely identify the image or to point torelated data.

In some implementations, a plurality of images is “quilted” together toform a mosaic or image composite. The each of the many images preferablyretains their unique watermark(s). These watermarks can be used toidentify or point to geospatial information. Or the watermark can pointto a description file that identifies the geolocation(s) of the depictedarea or image patch and/or provides additional information regarding thedepicted area.

An inventive aspect of the present invention utilities multiplewatermarks in an image to provide a user with geo-reference feedback.Consider the following example. A user pulls up an image on her computerfor display on her computer monitor, e.g., perhaps with the aid of imagehandling or viewing software. The image has been segmented into blocksand the blocks are embedded with a digital watermark to convey (or pointto) geocoordinates for the area depicted in the respective block. Thesegmentation is preferably imperceptible to the user. Each watermarkconveys or points to geo-coordinate information that is associated withthe block. A software plug-in (or a separate program) cooperates withthe image handling software and mouse (or touch screen) to provide userfeedback. As the user positions her mouse (e.g., as shown on-screen as astandard “arrow” or pointer), the user is presented with thegeo-coordinates associated with the subject location. The presentationcan take a myriad of forms ranging, e.g., from text, box-up boxes,graphics, etc. (From a more technical viewpoint, a mouse screen-locationposition is provided from the mouse (or mouse driver) to the operatingsystem, and then to the image handling software. The image handlingsoftware coordinates the screen-location with the displayed image. Adigital watermark corresponding to the pointed-to-area or block (i.e.,the mouse screen-location position) is decoded to obtain geo-coordinateinformation, and the geo-coordinate information is presented, perhapsafter accessing additional information from a database. As analternative, each digital watermark (or a group of digital watermarks)within an image is decoded once the image is pulled up on a monitor. Thewatermark identifier or payloads are associated with a particulardisplayed area. Then, when a user selects or points to a particulararea, the geo-coordinates can be displayed without first having todecode the digital watermark.

Instead of only presenting the user with geo-coordinates, the feedbackcan be significantly enhanced. Consider, for example, right-clicking themouse while the cursor is pointed to a particular image location. Theright click activates a pop-up window (or separate application, like amedia player or Windows Explorer, Internet browser, etc.). The pop-upwindow provides news broadcasts, if available, audio/video clip, relatedtext that is associated with the location pointed to by the mousecursor. From another perspective, the digital watermark embedded in theimage at the cursor location is decoded. The embedded watermark carriesor points to a geolocation. Information associated with the geolocation(or watermark identifier) is identified, e.g., from a database or newsource, and is then presented to the user via the window or mediaplayer.

(Of course, it should be appreciated that instead of a mouse cursor, atouch screen, touch pen, optical receptor screen (e.g., one activate bya laser pointer), etc. can be used instead of a mouse. Also, our use ofthe term “right click” is not limiting. Other conventional techniques ofactivating computer functionality can be suitably interchanged with a“right-click.”).

High-resolution images are huge in terms of the digital bytes requiredto represent the image. Often, such large images are down-sampled inorder to efficiently display or print image areas. Down sampling mayreduce the image resolution but it also reduces the file or area bytesize, making the down sampled image easier to handle. While downsampling provides a faster and perhaps easier way to handle images, itcan complicate the watermark detection process. Accordingly, in oneimplementation, we embedded a digital watermark at different resolutionswithin the image. Varying the number of pixels used for embedding awatermark component can achieve this “multi-scale” watermark. Forexample, for a high-resolution scale, a low number of pixels (maybe evenonly one pixel) is used to convey the watermark component; yet for alower-resolution scale, the same watermark component is conveyed over ahigher number of pixels, like a 6×6 block of pixels, 12×12 block, 16×16block, 20×20 block, 128×128 block of pixels, etc. Accordingly ourwatermark is detectable at multiple resolutions.

In another implementation, an embedded digital watermark includes anorientation component. The orientation component preferably provides areference used to determine a baseline or reference scale for the image.Once reconciled, however, the reference component provides clues todetermine the original resolution of the image.

Digital watermarks form the backbone in an image distribution system(FIG. 1). Satellite imagery is captured from a number of sources, e.g.,domestic (e.g., U.S. government), commercial and foreign (e.g., foreigngovernments). An image is communicated to any one of a number of datacenters 1-3 (e.g., corresponding to government, civil and non-governmentcenters). The image is digitally watermarked to include a uniqueidentifier. As discussed, the unique identifier may comprise ageo-location, capture time, or the identifier can be a serial numberthat is used to link to related information. The embedding preferablyoccurs prior to storage at a data center, so that the image can bestored and retrieved in the data center via the identifier.

A watermark provides a persistent identifier that is used to managerequests for imagery, as well as tracking and managing the distributionof such images to consumers 1-5. Consumers 1-5 may include governmentagencies, news and media, corporations, foreign entities, etc., etc. Ifdesired, every action (or a subset of actions) that is carried out onthe image can be reported to the data center for logging. For example,if an image is copied to a disk, such action is reported to the datacenter. Or if the image is cropped, scaled or printed, such is reportedand recorded in the data center—thus creating an audit trail for theimage. The audit trail can include user or device information as well.The image can be tracked via the identifier even as it is widelydistributed. From a system level, a security software module monitorsimages for a watermark embedded therein. The software module preferablyresides on each client system (e.g., computer terminal) in a closednetwork. Once detected, a watermark is decoded to obtain its identifier.The identifier is reported to a registry or data center, along with theparticular action which triggered the decoding (e.g., printing, storingto disk, editing, etc.).

The digital watermark also simplifies license compliance and reporting.With reference to FIG. 2, images are distributed to a number ofconsumers 1-5. The consumers can report image usage associated with thedigital watermark. (We note that a watermark can also help enforcelicensing terms, e.g., by providing copy or viewing restriction flags,by triggering reporting or activity, by limiting access based onenvironment (e.g., a secure computer or handheld device) or securityclearance, etc., etc. The watermark may also be used as a securitymeasure. The watermark can be used to carry security clearanceinformation, or ensure that the related image is not distributed in anunauthorized channel. Regulating software uses information carried bythe digital watermark to regulate access or enforce licensing terms,etc.). The digital watermark can also be used to link to complianceinformation, reporting forms and copyright notices, etc.

Our watermarks can be used in even more robust settings. Consider thesystem shown in FIGS. 3-7. An image is captured from an aerial platform(see FIG. 3). The captured image becomes “source data.” The source datais digitally watermarked to include a unique identifier as discussedabove. We note that a typical image capture method is illustrated inFIG. 3; that is, a satellite captures an image and then communicates thecaptured image to a ground station. The watermark embedding ispreferably carried out at the ground station or at an associated datacenter or registry. However, as we have discussed in our relatedapplications, the image-capturing platform itself (in the FIG. 3example, a satellite) can provide the watermark embedding.

The digitally watermarked source data (e.g., the captured image) isstored in a registry. The act of storing the source data in the registrypreferably triggers a searching process. (Alternatively, an automaticschedule or manual input triggers the searching process.) One objectiveof the searching is to find information that is related to the sourcedata, e.g., via geolocation or depicted subject matter. The types ofinformation are limitless, and can include related images, notes,reports, data, history, news from a broadcast, geographic informationsystem (GIS) layers and libraries, audio or video, imagery in libraries,geo-calibration data, periodicals and current or archived newspapers,current or historical weather, governmental and cultural information,consumer demographics, etc., etc. Searching can be facilitated in anumber of ways. For example, the search may include searching forinformation associated with the source data's watermark identifier. Thevalue of this searching method is particularly evident when thewatermark identifier is associated with a geolocation. In alternativesearching implementations the searching includes using the depictedgeocoordinates as searching criteria. In other implementations thesearching is facilitated by key words associated with depicted areas,associated political regions, associated cultures, depicted structuresand/or other information depicted in the source data. The key words canbe carried by a digital watermark. The searching becomes fullyautomated, since a digital watermark decoder can retrieve the key wordsfrom a watermark, and then communicate the key words to a search engineassociated with specific data libraries. Or suppose, for example, thatthe source data depicts the Washington Monument. The key word search mayinvolve searching databases, news sources, the internet, etc. forinformation related to the “Washington monument.” Or if the source datadepicts Boise, Id., a key word search may include the terms “Boise”and/or “Idaho.” (Of course, the search can include alternativetechniques such as topic searching, directed searching, etc., etc.). Therelated data, once identified, is associated in the data registryaccording to the unique identifier. Of course, instead of duplicatingthe related data, the data repository can include links to or addressesof the related data. Enhanced searching methods, such as those disclosedin assignee's U.S. patent application Ser. Nos. 09/636,102 and10/118,468 (published as US 2002-0188841 A1), can be implemented topopulate the registry as well.

The search can also extend into image libraries. Previously capturedimages, identified via geo-location or other referencing identifiers,can be associated with the source data as “related imagery.” Thisimage-centric search is shown in FIG. 5.

If the embedding is being carried out at ground stations, and not at thedata registry, the registry can serve as a unique identifier repository,to help ensure that identifiers do not collide. A ground station canquery the registry to obtain an appropriate identifier. Alsoconsecutively assigned identifiers can be pseudo-randomly or randomlyvaried.

The registry (or perhaps a match filter, as discussed below, or a clientsoftware plug in) can also serve as a watchdog or audit tracker. Awatchdog limits access to the source data and related data based onsecurity clearance, environment (e.g., whether the requesting party isin a secure facility or out in the field) or device type. A digitalwatermark can convey environmental limitations or required securityclearances that are associated with an image. The registry (or othermodule) compares the digital watermark information (e.g., indicatingsecurity clearance) with the requesting user's or computer terminal'sinformation (e.g., the user's or terminal's security clearance). Theimage or information is conveyed to user or terminal only if the user'sor terminal's clearance or environment is sufficient for the image orinformation.

An audit tracker serves to log movement and use of a watermarked image.The audit tracker can record access times, a person or terminalaccessing the watermarked image, security levels associated with usersor terminals accessing the watermarked image, printers at which theimage is printed, whether the image is locally downloaded or emailed,etc. The tracking is facilitated with the digital watermark identifier.Once decoded, the identifier is used to access a log or data record. Thelog is populated with entries as the image is handled, printed,downloaded, etc. Software resident on a user terminal and/or networkserver can be used to facilitate such monitoring and tracking.

At some point in the registry process, the source data preferablyundergoes a georeferencing (or photogrammetric) process (FIG. 6). Asdiscussed in this and in some of our above-identified relatedapplications, the source data undergoes an analysis that maps each imagepixel or block of pixels to a geolocation. In one implementation, thisprocess is used to derive a unique watermark identifier or to creategeo-coordinates for storage in the registry. The timing of this processis not critical, unless the watermark identifier directly depends onthis georeferencing. Otherwise the georeferencing can take place at anytime in the registry-populating process.

A match filter is used in connection with the registry (FIG. 7). Thematch filter preferably includes algorithms or protocols to directinformation to certain network nodes or terminals. Information can bedirected based on predetermined preferences or requests, or by subjectmatter. Or the match filter can serve as a search engine to allownetwork nodes to intelligently query the registry. (We note that in someimplementations, the registry is distributed and is mirrored as needed.The match filter can be similarly distributed). In one implementation,the match filter monitors data locations (e.g., such as databases,records, network sites or storage sites) that may include dataidentified by a watermark identifier or geo-location. New data receivedat these data locations can be tracked/recorded and optionally announcedor pushed to the interested parties. In another implementation, thematch filter filters information to users based on a user device thatwill receive the information. For example, the match filter maydetermine that a requesting device is a PDA (personal digital assistant)so the match filter sends a copy of the information that is compatiblewith the PDA. In this manner, the match filter can provide a contextsensitive filter. (The term “context sensitive” can also imply securityrestrictions. For example, while the PDA may be able to process andhandle a particular item, it may not be permissible to transmit the datato such a handheld device due to security concerns).

The match filter can optionally include (or cooperate with) an alertengine. The alert engine monitors the registry and sends out an alert(see FIG. 8) when new or updated information is received in theregistry. For example, the alert engine monitors which watermarkidentifiers (or geo-locations) are communicated to various networknodes. Then, if the registry receives new or updated information that isassociated with the identifier, the alert engine sends out anotification or alert to that node. The alert can be manifested in auser terminal/GUI via a pop-up window (e.g., FIG. 8), graphical icon(e.g., a blinking icon in a desktop window or control bar), email,instant message, etc. In another implementation, a network node orterminal schedules an alert request with the alert engine. For example auser may indicate that she would like a notification when updatedimagery arrives. The alert process can be relatively seamless for theuser. The user terminal (or alert engine) extracts the digital watermarkfrom an image that a user is currently viewing. The alert engine storesthis identifier as one to watch for updates. A registry flag (or otherindicator) that is associated with the identifier is set when an updatedimage is received into the registry. The alert engine recognizes the setflag and issues a notification to the user (or user terminal or networkaddress). Or the user can similarly request information based ongeo-location. The alter engine can also push fresh information (e.g.,recently captured imagery) to a network node or terminal. Or breakingnews (e.g., accessible via an internet link or audio/video/text messagestorage on a network site) can be similarly pushed to interestedparties.

Another inventive feature is to allow for removal of an embedded digitalwatermark from a digital image. For example, there may be some imageanalysis that requires the original, unwatermarked image. In oneimplementation, the digital watermark includes a payload. The payloadincludes a network address or a pointer that points to an original,unwatermarked image. The original, unwatermarked image is obtained withthe address or pointer. In another implementation, the digital watermark(or a registry record pointed to by the digital watermark) includeswatermark-embedding details (e.g., watermark gain, watermark embeddingprotocol, tiling data, and/or perceptual masking, etc., etc.), whichwill allow a watermark remover module to remove the embedded digitalwatermark. The watermark is decoded to obtain the identifier. Theidentifier is then used to instruct the watermark embedding informationto instruct the removal module in removing the digital watermark. Insome alternative implementations, the registry (or other auditingmodule) records image manipulation, if any, which occur as the image ishandled and analyzed. The manipulation data provides clues as to how theoriginal data was manipulated; thus, allowing the manipulation to bereversed. The embedding information and, optionally, manipulationinformation, helps facilitate the near-perfect removal of the watermark.(In an alternative implementation, the watermark is embedded accordingto a predetermined rule or protocol, and the removal module removes thewatermark according to the predetermined rule or protocol. In othercases we employ a so-called “reversible” watermarking technique, as,e.g., discussed in assignee's U.S. patent application Ser. Nos.10/319,404 (published as US 2003-0149879 A1), Ser. No. 10/319,380(published as US 2003-0179900 A1), and PCT application no.PCT/US02/40162, published as WO 03/055130).

As indicated, the watermark(s) can identify the imaging system, thedate/time of data acquisition, satellite ephemeris data, the identity ofintervening systems through which the data passed, etc. One or morewatermarks can stamp the image with unique identifiers used insubsequent management of the image data, or in management of meta dataassociated with the image.

Seal of Completeness

There are additional benefits in creating a georeferenced image registrysystem using digital watermarks. Consider the following. A classicnotion in most industries is a “stamp” or “seal” or a similar concept toindicate that some object has successfully completed its appointedrounds. Call it branding, call it formality, or call it a soft form of“authenticity;” the historical momentum behind such a branding conceptis huge. In one embodiment, to evidence that a given image has beenproperly georeferenced (under a chosen standard) and/or registered in adatabase, the image is digitally watermarked. The digital watermarkprovides a formalized “seal of approval.” The digital watermark itselfbecomes a seal. In one embodiment, a watermark identifier is obtainedfrom an online repository, which issues and tracks authenticidentifiers. The repository can be queried to determine the date andtime of issue. Or the identifier can be linked to a seal or companylogo. Software and/or hardware are configured to routinely read embeddeddigital watermarks and display an appropriate brand logo, seal, orcertification. The “seal” itself then becomes a functional element of astandardization process, serving many functions including permanentattachment to standardized and dynamic metadata.

Geographic Information System

Digital watermarking techniques can be advantageously combined withgeographic information systems (“GIS”). GIS combines “layers” ofinformation about a given geographic location to provide a betterunderstanding of that location. What layers of information are combineddepends on the purpose, such as finding the best location for a newstore, analyzing environmental changes or damage, topology, looking atthe acoustic impact to a given area, outlining roads or buildings,evaluating elevation, viewing similar crimes in a city to detect apattern, and so on. GIS provides tools to query, analyze and map datacorresponding to a given spatial location.

Most commonly, a GIS system includes a general-purpose computerexecuting GIS software. The GIS software provides functions and tools tostore, analyze, and display (e.g., via a graphical user interface)information that is tied to a given geo-spatial location. For example,GIS software may include tools for entering, analyzing, queering and/ormanipulating layer information such as roads, buildings, rail roads,shipping lanes, image scale, water resources, vegetation, demographics,land use, cities, city infrastructure, schools and hospitals, rivers orother water bodies, petroleum and natural resources, political lines,cultural information, geographic information, military installations,air lanes, country boundaries, etc., etc., for the given spatial area.The layers may be integrated with or displayed over a base map (perhapsanother layer itself) or aerial image. In such cases the layers may bedisplayed over the image, e.g., as graphics or colors, lines or boxes,colored areas, arcs, lines, points, 3-D effects, images, etc., etc. Inother cases the layers may be displayed separately or with only aportion or outline of the base map or image. As a simplified butillustrative example, FIG. 9 a shows an aerial image depicting a riversection. FIG. 9 b illustrates one possible layer that is associated withthe FIG. 9 a image. The FIG. 9 b layer outlines the river's shoreline asdepicted in FIG. 9 b. A GIS system may display the FIG. 9 b layerseparate from the image, or may “overlay” the FIG. 9 b layer (e.g.,represented in a color outline, etc.) on top of the FIG. 9 a image.Other layers for the FIG. 9 a image may include, e.g., a historical viewof the shoreline, e.g., as of 100 years ago (helpful in showing rivermigration or erosion); airport or road positions, population density;and/or fish habitats, etc, etc.

GIS software typically includes (or cooperates with) a databasemanagement system. The database helps maintain layer information. MostGIS systems include a graphical user interface (GUI) to display layerinformation, perhaps displayed over an image or base map. GIS layerinformation can be presented via the GUI in various formats. GISsoftware is available from, e.g., ESRI (e.g., ArcView), Erdas (e.g.,Imagine), Sensor System (e.g., RemoteView), 3DI Geographic Technologies,Apic, RMSI, AutoDesk MapGuide, Mapinfo MapXtreme, ArcIMS (Internet MapServer) etc., etc. Further background information regarding GIS can befound with reference to, e.g., U.S. Pat. Nos. 6,542,813 and 6,389,356;U.S. Published Patent Application Nos. 20030014185 and 20020145620 (suchpatent documents are herein incorporated by reference); and on theinternet at, e.g., www.gis.org and www.gis.com.

GIS systems typically organize layer information using two primarytechniques—raster data and vector data. Raster data provides storage,processing and display of spatial data. A depicted area is divided intorows and columns, which form a grid structure. Cells within this gridstructure contain an attribute value as well as location co-ordinates.The spatial location of each cell is implicitly conveyed by the orderingof the grid, unlike a vector structure that stores topology explicitlyVector data includes lines and/or arcs, defined by beginning and endpoints, which meet at nodes. The locations of nodes and the topologicalstructure are typically stored explicitly. Features are usually definedby their boundaries and curved lines are represented as a series ofconnecting arcs. A vector data example is a file including propertyboundaries for a housing subdivision.

GIS layer information is registered, or correlated to, a correspondingspatial location on an image (or map). (Layer data may also undergoprojection conversions, for example, that integrates a layer into a GIS.Projection is a mathematical technique used to transfer information fromthe Earth's three-dimensional curved surface to a two-dimensional mediumsuch as a computer screen). We have found that digital watermarks can beused for GIS layer registration (or spatial alignment with an image).For example, a digital watermark may convey or identify an imageposition that should correspond with a particular raster cell (e.g., thefirst cell) or with a vector node (or beginning or ending point). Or adigital watermark can provide other orienting components to help alignor register layer information onto or with a corresponding image. Insome implementations, layer information (or a layer itself) isassociated with a particular image via a digital watermark identifier.Returning to the FIGS. 9 a and 9 b example, the FIG. 9 a image mayinclude a digital watermark embedded therein. The digital watermarkincludes an identifier, which is used to interrogate a GIS database tofind the FIG. 9 b layer. The digital watermark may even conveyorientation information to help spatially register the layer with theimage.

Now consider a situation where an image is manipulated or distorted. Forexample, the image is scaled, rotated and/or cropped. Any GIS layerinformation that was once registered is now likely out of alignment withrespect to the manipulate image. An improvement is to use a digitalwatermark to effectively reregister the GIS layer information to themanipulated image. In a first implementation, we use a so-calledorientation component carried by the digital watermark to determine therelative scale of the manipulated image (see, e.g., assignee's U.S.patent application Ser. No. 09/503,881 and U.S. Pat. No. 6,122,403,which are each herein incorporated by reference for an ever furtherdiscussion of watermark-based orientation). Geo-coordinate informationis retrieved from a watermarked (but manipulated) image block toidentify the depicted image block location. Decoding several of suchwatermarked image blocks, and knowing the relative scale from theorientation component, can be used to help accurately reregister thelayer information. In another implementation, at least one digitalwatermark component is analyzed to determine the scale of themanipulated image. The GIS layer information is adjusted accordingly. Instill another implementation, decoded digital watermark information isused to identify a location(s) in the manipulated image, and GIS layerinformation corresponding to the location(s) is mapped (or projected) tothe identified location(s). Accordingly, we use a digital watermark toreregister GIS layer information when an image is manipulated. In otherimplementations, the digital watermark decoder recognizes via theorientation component that an image includes some distortion. The imageis realigned prior to GIS layer overlay. In other implementations thedigital watermark will include a spatial position indicator. A GIS layercan be aligned with an image based on the position indicator. Forexample, if the image is cropped, and a particular GIS layer based itsregistration to the image on the image's left upper corner, the layerregistration would be lost. However, if a digital watermark conveys aspatial position indicator, the layer can base its registration from thewatermark position indicator. Sometimes a spatial position indicatorwill be the presence of a watermark in a given image region (e.g., awatermark is embedded in a region depicting a lake or road). Other timesthe spatial position indicator will be conveyed through a watermarkpayload, perhaps identifying the spatial position indicator as adistance from an image object. Of course other spatial positionindicators can be conveyed via a digital watermark as well.

In another implementation, we provide a digital watermark decodingplug-in, which cooperates with a GIS software program. The plug-insearches an aerial image for a digital watermark embedded therein. Thedigital watermark includes an identifier. Once decoded from the digitalwatermark, the identifier is passed to the GIS software (or perhapsdirectly to a database), which accesses a GIS database including aplurality of information layers. A layer (or a set of layers) associatedwith the digital watermark is identified with the identifier. The layer(or set of layers) is provided to the GIS software for use with theimage. If the watermark includes a registering component, the layer isaligned or registered with the image via the registry component.

Persistent Information Notation

One aspect of the present invention is a Persistent Information Notation(or “PIN”) that is embedded in an image. We liken our PIN to anelectronic sticky note. The PIN links the watermark to a registrationdatabase. Once decoded, the PIN is used to interrogate the registrationdatabase to locate, e.g., related metadata. Watermarked imagery is madeavailable for display on an image analyst's computer workstation. Duringdisplay, the analyst can access and append additional imagery andtextual information relevant to the currently displayed scene—stored inthe database—via the watermark embedded PIN. The watermark is preferablyrobust. For example, the PIN survives even if the watermarked imagery isprinted and scanned back into digital form. The PIN can be used togather disparate or disassociated image-related information (e.g., bystoring them in a data record associated with the PIN), and appendrelevant information to the watermarked image or image chip fordissemination to downstream analysts (e.g., by storing the relevantinformation to be accessible via the PIN).

A PIN implemented with digital watermarking provides the capability toadd identifying information and link dynamic information to an image andits derivative images in a robust fashion and with increasedsurvivability.

Illustrative Scenario Involving a PIN

Upon initial analysis a first image analyst adds commentary to an image.The commentary resides in a database, which is linked to the image viaan embedded watermark PIN. As the image continues in its lifecycle, asecond analyst is provided an image chip (e.g., an image segment or aderivative from the image) corresponding to the image. The secondanalyst accesses related information, including the original image, viathe embedded watermark PIN. The second analyst also appends and updatesthe annotations in the database record that is associated with the PIN.Next, a printed image is generated and passed to a third analyst forfurther hard copy exploitation. This printed image is not delivered witha complete set of notes. Using a scanner or input imaging device (e.g.,a PC camera, digital camera, flatbed scanner, etc.), the third analystreads the digital watermark (and PIN) from the image, and links to thedatabase via PIN. (See, e.g., assignee's U.S. Pat. Nos. 6,324,573,6,371,214, 6,286,036, 6,122,403 and 5,841,978 and U.S. patentapplication Ser. No. 09/571,422, filed May 15, 2000, which are eachherein incorporated by reference, for various linking techniques). Thus,the third analyst gains full knowledge of the image-relevant informationthat she is cleared to access. (In this regard, the analyst may need topresent her security level credentials prior to image access. The PINmay include a required security clearance needed to access the relatedinformation. If the security level credentials match the PIN's securitylevel, access is allowed). Finally, a fourth analyst wants to use theimage and associated comments for producing a report. However, loss ofimage header information makes it impossible to determine the imagesource, location, or the age of the image. The fourth analyst decodesthe watermark PIN in a digital copy of the image (e.g., perhaps throughan automated process initiated via right-clicking on a displayed copy ofthe image). The PIN is used to link to the related information. Thelinked information is used to access the stored information. The stored,related information may include other image data relevant to the image;latitude and longitude of the source image; a list of other images andreports related to the source image. The report can now be generated.

Some conventional systems for accessing imagery and related informationrequire that an image have “attached” to it some identifyinginformation. In many situations the attached information is easilyseparated from the image; therefore, identification of the imagery,retrieval of its ancestors, and retrieval of related text information,is difficult or even impossible. Our techniques embed a digitalwatermark component in an image as a persistent image identifier andthus mitigate impact due to notation loss, change in file format, orloss of header data.

(We note that while we use the term “PIN” above to represent apersistent identifier, the term PIN can also be used to representinformation stored in a database that is associated with a persistentidentifier).

CONCLUSION

The foregoing are just exemplary implementations of the presentinvention. It will be recognized that there are a great number ofvariations on these basic themes. The foregoing illustrates but a fewapplications of the detailed technology. There are many others.

Some watermarks used in the foregoing embodiments can be “fragile.” Thatis, they can be designed to be lost, or to degrade predictably, when thedata set into which it is embedded is processed in some manner. Thus,for example, a fragile watermark may be designed so that if an image isJPEG compressed and then decompressed, the watermark is lost. Or if theimage is printed, and subsequently scanned back into digital form, thewatermark is corrupted in a foreseeable way. (Fragile watermarktechnology is disclosed, e.g., in application Ser. Nos. 09/234,780,09/433,104 (now U.S. Pat. No. 6,636,615), Ser. No. 09/498,223 (now U.S.Pat. No. 6,574,350), Ser. Nos. 09/562,516, 09/567,405, 09/625,577 (nowU.S. Pat. No. 6,788,800), and Ser. No. 09/645,779 (now U.S. Pat. No.6,714,683). Each of these patent applications is herein incorporated byreference.) By such arrangements it is possible to infer how a data sethas been processed by the attributes of a fragile watermark embedded inthe original data set.

To provide a comprehensive disclosure without unduly lengthening thisspecification, applicants incorporate by reference, in their entireties,the disclosures of the above-cited U.S. patents and applications. Theparticular combinations of elements and features in the above-detailedembodiments are exemplary only; the interchanging and substitution ofthese teachings with other teachings in this application and theincorporated-by-reference patents/applications are contemplated.

There are many embodiments discussed herein which may benefit from theinclusion of two different watermarks. For example, a first watermarkmay include information evidencing (or pointing to) georeferencinginformation, while a second watermark includes a database identifier orlocation. The second watermark may alternatively include (or pointtoward) information pertaining to events, people or animals identifiedin the photograph, occasions, groups, institutions, copyright ownership,etc. Or an image may include both a robust watermark and a copy-tamperfragile watermark.

Also, while the presently preferred embodiments have focused on images,the present invention is not so limited. Indeed, audio and video can beembedded with geolocation information to identify the captured (orbroadcast) location and, in the case of video, a location depicted inthe video. The system shown in FIGS. 3-8 can be adapted to handle suchgeolocation embedded audio and video.

Although not belabored, artisans will understand that the systemsdescribed above can be implemented using a variety of hardware andsoftware systems. One embodiment employs a computer or server with alarge disk library, and capable database software (such as is availablefrom Microsoft, Oracle, etc.). The registration, watermarking, and otheroperations can be performed in accordance with software instructionsstored in the disk library or on other storage media, and executed by aprocessor (or electronic processing circuitry) in the computer asneeded. (Alternatively, dedicated hardware, or programmable logiccircuits, can be employed for such operations).

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only and should not be taken aslimiting the scope of the invention.

1. A method of registering an image captured from an aerial platform inan image registry, the image comprising a digital watermark embeddedtherein, the digital watermark comprising an identifier, the digitalwatermarking embedded in the image through relatively slight alterationsto data representing the image said method comprising: storing the imagein the registry so as to be retrievable based on the identifier;automatically searching computer databases for information associatedwith the identifier; and upon finding additional information that isassociated with the identifier, associating the additional informationwith the image via the digital watermark identifier.
 2. The method ofclaim 1 wherein the identifier comprises a geo-location associated withan area depicted in the image.
 3. The method of claim 2 wherein theadditional information comprises information associated with thegeo-location associated with the area depicted in the image.
 4. Themethod of claim 1 wherein the identifier comprises a key word, and saidsearching comprises providing the key word to a search engine associatedwith the computer databases.
 5. A method of operating a network filter,the network comprising a plurality of network nodes, the network filterto help manage images or video each including an identifier embeddedtherein in the form of a digital watermark, said method comprising:monitoring image or video traffic to a particular network node,identifying a first image or video at the particular network nodethrough decoding a digital watermark embedded in the first image orvideo, the digital watermark including a first identifier; determiningadditional information associated with the first image based at least inpart on the first identifier; and notifying the network node of thepresence of the additional information.
 6. The method of claim 5 whereinthe additional information comprises a recently captured image or videocorresponding to the first image or video, the recently captured imageor video comprising a second identifier that is associated with thefirst identifier.
 7. The method of claim 6 wherein the first identifierand the second identifier represent geo-coordinates.
 8. The method ofclaim 5 further comprising: upon receipt of a request, providinginformation associated with the request, wherein the request comprisesan image or video identifier.
 9. A method of scheduling an alert toannounce the presence of available information associated with an imageor video, wherein the image or video includes at least one digitalwatermark embedded therein, the digital watermark comprising anidentifier, said method comprising: receiving an identifier from anetwork terminal, wherein the network terminal obtains the identifierthrough decoding a digital watermark from an image or video; monitoringdata records associated with the identifier; and upon identifyingupdated information associated with the identifier, sending a message tothe network terminal announcing the availability of the updatedinformation.
 10. The method of claim 9 wherein said monitoring comprisesdecoding digital watermarks from data stored in the data records. 11.The method of claim 9 wherein the identifier comprises geo-locationinformation.
 12. The method of claim 9, wherein the updated informationcomprises aerial imagery or video.
 13. The method of claim 9 wherein thedigital watermarking utilizes a transform domain.
 14. A method ofcorrelating a geographic information system (GIS) layer with an image orvideo comprising: decoding a digital watermark embedded in the image orvideo, the digital watermark comprising at least a plural-bitidentifier; identifying a GIS layer which is associated with the imagevia the plural-bit identifier; and providing the GIS layer.
 15. Themethod of claim 14 wherein the GIS layer is spatially registered to theimage or video via the digital watermark.
 16. The method of claim 15wherein the digital watermark comprises an orientation component and theGIS layer is spatially registered to the image or video via at least theorientation component.
 17. The method of claim 15 wherein the digitalwatermark comprises an orientation component, and the orientationcomponent is utilized to determine a relative spatial positionassociated with the image or the GIS layer.
 18. The method of claim 14wherein the digital watermark utilizes a transform domain.
 19. Acomputer-readable medium comprising executable instructions storedtherein, the instruction comprising instructions to carry out the methodrecited in claim
 16. 20. A computer-readable medium comprisingexecutable instructions stored therein, the instruction comprisinginstructions to carry out the method recited in claim 17.